European COVID-19 Forecast Hub Evaluation Report for UMass-MechBayes
2021-10-11
Forecast visualisation
Forecasts of cases/deaths per week per 100,000. The date of the tab marks the date on which a forecast was made (only the latest forecasts and the previous 4 weeks shown).
2021-10-11
Cases
No forecasts available, possibly because of recent anomalies in the available data.
Deaths
2021-10-04
Cases
No forecasts available, possibly because of recent anomalies in the available data.
Deaths
2021-09-27
Cases
No forecasts available, possibly because of recent anomalies in the available data.
Deaths
2021-09-20
Cases
No forecasts available, possibly because of recent anomalies in the available data.
Deaths
Predictive performance
Scores separated by target and forecast horizon.
Cases
1 week ahead horizon
Overall skill
Skill is shown as relative absolute error; weighted interval score is not calculated as the 23 quantile levels required are not being provided by the model. Scores are only created for models that were used for forecasts in each of the last 4 weeks, excluding periods during which there were anomalies in the data. At the moment the model does not fulfill that criterion.
Skill over time
Skill is shown as weighted interval score is not calculated as the 23 quantile levels required are not being provided by the model.
Plots are only created for models that were used for forecasts in each of the last 4 weeks, excluding periods during which there were anomalies in the data. At the moment the model does not fulfills that criterion.
2 weeks ahead horizon
Overall skill
Skill is shown as relative absolute error; weighted interval score is not calculated as the 23 quantile levels required are not being provided by the model. Scores are only created for models that were used for forecasts in each of the last 4 weeks, excluding periods during which there were anomalies in the data. At the moment the model does not fulfill that criterion.
Skill over time
Skill is shown as weighted interval score is not calculated as the 23 quantile levels required are not being provided by the model.
Plots are only created for models that were used for forecasts in each of the last 4 weeks, excluding periods during which there were anomalies in the data. At the moment the model does not fulfills that criterion.
3 weeks ahead horizon
Overall skill
Skill is shown as relative absolute error; weighted interval score is not calculated as the 23 quantile levels required are not being provided by the model. Scores are only created for models that were used for forecasts in each of the last 4 weeks, excluding periods during which there were anomalies in the data. At the moment the model does not fulfill that criterion.
Skill over time
Skill is shown as weighted interval score is not calculated as the 23 quantile levels required are not being provided by the model.
Plots are only created for models that were used for forecasts in each of the last 4 weeks, excluding periods during which there were anomalies in the data. At the moment the model does not fulfills that criterion.
4 weeks ahead horizon
Overall skill
Skill is shown as relative absolute error; weighted interval score is not calculated as the 23 quantile levels required are not being provided by the model. Scores are only created for models that were used for forecasts in each of the last 4 weeks, excluding periods during which there were anomalies in the data. At the moment the model does not fulfill that criterion.
Skill over time
Skill is shown as weighted interval score is not calculated as the 23 quantile levels required are not being provided by the model.
Plots are only created for models that were used for forecasts in each of the last 4 weeks, excluding periods during which there were anomalies in the data. At the moment the model does not fulfills that criterion.
Deaths
1 week ahead horizon
Overall skill
Skill is shown as weighted interval score.The table shows the skill of the UMass-MechBayes model (Model skill), the ensemble model (Ensemble skill), and the best individual model (Best skill).
Skill over time
Skill is shown as weighted interval score.
2 weeks ahead horizon
Overall skill
Skill is shown as weighted interval score.The table shows the skill of the UMass-MechBayes model (Model skill), the ensemble model (Ensemble skill), and the best individual model (Best skill).
Skill over time
Skill is shown as weighted interval score.
3 weeks ahead horizon
Overall skill
Skill is shown as weighted interval score.The table shows the skill of the UMass-MechBayes model (Model skill), the ensemble model (Ensemble skill), and the best individual model (Best skill).
Skill over time
Skill is shown as weighted interval score.
4 weeks ahead horizon
Overall skill
Skill is shown as weighted interval score.The table shows the skill of the UMass-MechBayes model (Model skill), the ensemble model (Ensemble skill), and the best individual model (Best skill).
Skill over time
Skill is shown as weighted interval score.
Forecast calibration
The table and plot below show this week’s coverage of the ensemble model at the 50% and 95% level, across the 32 countries. This shows the proportion of observations that fall within a given prediction interval. Ideally, a forecast model would achieve 50% coverage of 0.50 (i.e., 50% of observations fall within the 50% prediction interval) and 95% coverage of 0.95 (i.e., 95% of observations fall within the 95% prediction interval). Values of coverage greater than these nominal values indicate that the forecasts are underconfident, i.e. prediction intervals tend to be too wide, whereas values of coverage smaller than these nominal values indicate that the ensemble forecasts are overconfident, i.e. prediction intervals tend to be too narrow.
Overall coverage
PIT histograms
The figures below are PIT histograms for the all past forecasts. These show the proportion of true values within each predictive quantile (width: 0.1). If the forecasts were perfectly calibrated, observations would fall evenly across these equally-spaced quantiles, i.e. the histograms would be flat.
